DOCUMENT RESUME 



ED 359 265 



TM 020 110 



AUTHOR 
TITLE 

PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



Mendro, Robert 

The Place of Process Evaluation in Action 

Research* 

Apr 93 

10p.J Paper presented at the Annual Meeting of the 
American Educational Research Association (Atlanta, 
GA, April 12-16: 1993) . 

Reports - Evaluative/Feasibility (142) — 
Speeches/Conference Papers (150) 

MF01/PC01 Plus Postage. 

^Action Research; Curriculum Development; ^Curriculum 
Evaluation; Educational Assessment; ^Educational 
Research; Elementary Secondary Education; Evaluation 
Methods ; "Formative Evaluation; ^Research 
Methodol ogy ; "Social Science Research; Student 
Participation; Time Factors (Learning) 



ABSTRACT 

Process evaluation is a critical element in 
understanding the results of any research. Because action research is 
conducted by practitioners, there is a tendency to think that there 
is little need for process evaluation. Yet, the complexity of factors 
underlying most educational treatments tends to defeat all but the 
best memories when attempting to recall specific facts regarding the 
treatment. Hence, there is a ciear need for nrocess evaluation in all 
but the simplest of action research undertakings. In this paper, the 
emphasis is on fundamental techniques that would underlie the 
evaluation of an investigation into implementation of a curriculum 
intervention. Four aspects of conducting process evaluations of 
curriculum intervention are considered. The first aspect is defining 
the actual curriculum as opposed to the intended curriculum. The next 
aspect is considering and tracking the level of student involvement. 
Thirdly, the process of instruction, the use of assessment in 
instruction, and the relation of both to process evaluation are 
examined. A final consideration is tracking time by instructional 
element through process evaluation. Each of these aspects must be 
examined in a carefully implemented process evaluation, an effort 
that will mere than repay the practitioner. (SLD) 
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THE PLACE OF PROCESS EVALUATION IN ACTION RESEARCH 

Robert Mendro 
Dallas Independent School District 

In general, one of the mo< important elements of research in education and the 
social sciences is process evaluation. Part of the reputation of research in education as 
contradictory or even slipshod stems from the relative lack of control and understanding 
of the research paradigm through adequate process evaluation. Consider the following 
hypothetical exampie. 

Mathematics-education authority X proposes a new way of teaching the solution of 
higher-order equations. It is based on the latest research and X has prepared a thorough 
curriculum incorporating nothing but the best in techniques and content. A research 
project is proposed and implemented by a dozen teachers in three forward-looking 
districts. After an extensive two-day in-service on the intricacies and joys of the new 
method, it is launched with great expectations. Six cleverly selected control classes are 
chosen from the same districts. The makeup of the control classes mirrors the student 
population of the experimental group almost perfectly. The control classes are to 
implement the "old-fashioned" curriculum (miraculously, all districts, control and 
experimental, use the same textbook adoption) while the project classes implement the 
brave new approach. At the end of the trial, all the students are given a carefully crafted 
criterion-referenced test, with several perfo.mance items (of course) to insure that the test 
is an authentic measure. 

The results come in. Contrary to all expectations they are both self-contradictory 
and at variance with those in the literature. Four experimental and one control class (with 
perversely different population characteristics) show great test results, five experimental 
and three control classes have only middling scores, and the remainder do poorly by any 
measure. Overall, the experimental classes have a slight edge in test results, but after an 
exhaustive analysis of the data with an impressive array of statistical might, our authority, 
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X, who is nothing if not conscientious, is forced to conclude that the experimental 
treatment just fails to have the right stuff. 

Regrettably, our friend has neglected to do a thorough job of evaluating the trial. 
Specifically, of the twelve experimental classes, only three (two which were in the top- 
scoring group) came close to implementing the program as conceived and presented in the 
in-service sessions. The others deviated by various degrees along a number of dimensions. 
Severai implemented what looked suspiciously like the old curriculum with fancy new 
clothes. Of the control group, three classes came somewhat close to implementing the old 
curriculum, another one implemented the teacher's own method of coming up with 
solutions of equations, one implemented a program that had many elements of the 
experimental program, and one implemented hardly any program at all. Our authority has 
based the carefully analyzed conclusions of the study on the implementation of many more 
programs than were contemplated in the original design of the study or his universe. 

While this example over-simplifies the situation in much educational research, (and 
may also have given the experiment many respectable features lacking in a number of 
studies) it illustrates a common problem. Much educational research fails to adequately 
describe, understand, and, often, control the actual processes under stu^y. As the 
necessity for doing research is increasingly promulgated for a larger number of 
practitioners under the aegis of action research, process evaluation is no less important. 

In action research, an individual practitioner or a group of practitioners is likely to 
have more immediate contact with the research being done since the practitioner is usually 
the implem^nter as well as the researcher (but no. always). This will result in more 
awareness of the events transpiring in the process of implementing an experimental 
program, but does not guarantee the accumulation of the type of knowledge necessary to 
answer questions about why specific results were or were not attained. 

Yet, action research offers perhaps the best opportunity to collect valuable process 
information with which to make detailed analyses of the workings of a program or 
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experimental procedure. When the researcher and implementer are one and the same, t* y 
have the luxury of having time to record what occurs in the trial. (Of course, some 
practitioners will claim, often with considerable justification, that they have less time than 
needed precisely because they are the implementers.) 

The critical issues for building better process evaluation in action research are 
increasing awareness of the payoff inherent in process evaluation for the participants in 
action research projects, exposing these participants to some of the types of activities 
which constitute process evaluation, helping them to build effective ways of undertaking 
process evaluation, and assisting them in understanding the results of process evaluation 
and the relationship of the results to the research being conducted. This paper will focus 
on the second of these issues. It will be further limited to process evaluation associated 
with research conducted in the classroom to improve instruction. 

Other papers in this symposium deal with the problems of using data to identify 
student and school strengths and weaknesses and to set goals in an instructional program. 
These analyses help focus both the research done to improve instruction and serve as a 
gauge of whether the research is successful or not. The process of instruction is the 
means by which these goals are accomplished. The elements of classroom instruction 
which will be delineated are fairly straightforward: (1) the curriculum used for instruction; 
(2) the degree to which the entire class is involved in the instructional process; (3) the 
instructional process, including on-going assessment of progress; and, (4) the time spent 
on instruction and practice. Naturally, the evaluation of these elements will have much 
overlap among them. 

Curriculum . One of the most easily overlooked elements in the instructional process 
is the curriculum. Curriculum will be defined, simplistically, as what teachers teach. (The 
author once heard another researcher attribute to Barak Rosenshine the pithy summary of 
Rosenshine's research on teaching as "Kids learn what teachers teach." That felicitous 
phrase, while being more true than would be generally acknowledged, serves as a good 
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general guide to conducting process evaluation.) Among the sources for the curriculum 
may be textbooks, instructional guides and outlines field trips, tests, the teacher's own 
experience, or other students. Guides to the process evaluator in determining what 
constitutes the sctual curriculum are examination of lesson plans, examination of posted 
and filed student work, and observation of lessons. 

The critical element is to focus attention on what has and, as important, what hasn't 
been taught. No one guide is generally sufficient for an external evaluator conducting 
process evaluation. For example, lesson plans may say that main idea or problem solving 
is being taught daily. An examination of student work may indicate that the only place it is 
occurring is in the verbal exchanges between the students and the teacher, or, possibly, 
only in the lesson plans. Thus, for a practitioner engaged in action research serving as an 
external evaluator for another participant, verification through multiple sources is an 
essential activity. 

When the process evaluator and the practitioner are one and the same, as is often 
the case in action research, the most effective approach to process evaluation is to draw 
the practitioner's attention to what actually was taught to whom. For example, a teacher 
conducts a reading lesson from a text and at the end of the lesson asks the questions in the 
teacher's guide. Techniques such as logs of teaching activities will serve as a record of the 
activities in the lesson and a summary. A chart of students responding to the questions 
will complete the other half of the information collected. The analysis of the log and 
response chart at another time will help focus on the elements in the actual curriculum 
and, incidentally, on its delivery. The end result of this type of analysis of the curriculum 
is often that kids did not learn something because it was rarely or never taught. Indeed, a 
number of process evaluation studies conducted in Dallas show that the primary barrier to 
increased test performance on higher-order thinking skills was the fact that they were 
rarely taught and, when taught, only a small subset of students were typically engaged in 
learning. 
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Implicit in the examination of curriculum is either a pre-analysis of the intended 
curriculum or frequent periodic analyses of what was taught that not only tracks content, 
but also what level the content was designed to be or actually taught at. For example, 
analysis of subject by student responses may indicate that all students are being taught a 
given skill while an analysis by taxonomic level may indicate that it was never taught 
above the first level: knowledge. 

Who was taught . An essential element of instruction is assuring that all students are 
taught. A critical part of process evaluation is to track who was taught and how much 
was taught to each student. In the example presented earlier, a response log was used to 
determine who answered what questions and to track the nature of the responses. A 
grade book tracking student work and test results, progress charts, records of individual 
help provided, charts of help provided during guided practice, and other such devices will 
serve here. Where they are affordable, computer-based management systems can be 
effective tools in tracking students as part of the process of conducting process evaluation 
of the curriculum. 

The hardest part for most practitioners will be tracking student participation and 
teacher-student interactions by subject matter. The latter qualification is important since 
an analysis of who responded without knowing to what will give the teacher and any other 
practitioner/researchers much less critical information than an analysis of response by 
subject matter. As indicated earlier, the analysis should also track the taxonomic level of 
the subject matter. 

Assessment . One of the best tracking tools of who was taught what is the use of 
frequent formal and informal measurement. As other papers presented in this symposium 
have or will indicate, frequent measurement is both a guide to what was taught and the 
success of the methods with each student. To be simplistic again, education can be 
reduced to nothing more than a continuous cycle of teaching and assessment. This also 
serves as the basis for effective process evaluation since interim assessmem is a productive 
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method for tracking implicit teaching. Further, when all other evidence indicates 
something was included in the curriculum and assessment indicates that it was not learned, 
the process evaluation can point to how it was taught and often provide the most 
promising direction for further investigation. 

Assessment, as indicated, does not have to be formal. Continual informal 
assessment will provide as much and often more information than formal assessment. (In 
fact, with effective instructional delivery, formal assessment is rarely more than a 
confirmation of information already well in hand.) The problem for the process evaluator 
or the practitioner will be the effective tracking of responses. Generally, this should be in 
written form. Many practitioners will experience overload trying to track the results of 
continual informal assessment without some sort of written record. However, this is not 
intended to denigrate regular assessment of interim comprehension for internal guiding of 
a lesson. For most practitioner/evaluators, frequent whole-class responses, where the 
response is clearly indicated by student, will give an ongoing sense of the success of 
instruction. 

Time . As numerous researches have suggested, time is one of the most critical 
elements of instruction. Systematic tracking of the time devoted to each part of 
instruction can be illuminating when combined with the other elements of process 
evaluation. A practitioner may be implementing an instructional procedure with all of the 
students, following the outlines of the procedure, and assuring that, when implemented, 
several levels of the taxonomy are assessed. But the practitioner may be failing to give 
students sufficient time to understand the concept or enough time to practice and absorb 
the details of the concept. 

Here, analysis of the curriculum before starting the course of action research or 
constantly reviewing the results of the process evaluation during the research will be of 
considerable help in determining relative time allotments for each aspect of the curriculum. 
The continual reassessment will indicate which outcomes are to be most emphasized and 
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which are receiving what proportion of time. Further, such a course of action will help 
clarify relative goals for each of the practitioners. 

In terms of tracking time, many methods can be employed. The simple use of a 
stopwatch will allow the easiest tracking of elapsed time for each activity. Charting 
beginning and ending times is straighforward but more tedious than the use of a 
stopwatch. If a timeclock is available, punching timecards is an easy way to track time. 
An added advantage of all these procedures is that they tend to make students aware of 
both the time allotted for and the relative importance of each element of the curriculum. 

Summary . A critical element in understanding the results of any research is detailed 
process evaluation. Because process evaluation is labor intensive, it is not routinely 
planned for and implemented in much of educational research. In action research, there is 
a tendency to think that since practitioners will be conducting the research, there is little 
need for process evaluation. Yet, the complexity of the number of factors underlying most 
educational treatments tends to defeat all but the best of memories when attempting to 
recall specific facts regarding the treatment. Hence, there is a clear need for process 
evaluation in all but the simplest of action research undertakings. Effective process 
evaluation also has the added advantage of increasing practitioner awareness of the types 
of variables critical to success in an educational endeavor. 

Process evaluation, when implemented propuriy, is a complex and demanding 
activity, if for no other reason than most of the educational activities evaluated are much 
more complicated that usually recognized. This paper has been strictly limited to 
describing some of the fundamental techniques which would underlie the evaluation of an 
investigation into the implementation of a curriculum intervention. The techniques 
described would also be useful if the goal of the action research was simply to investigate 
how *be current curriculum was being implemented. As a result, the paper has avoided 
process evaluation methods for other types of research that might be implemented. 
Furt'ier, it has not considered several important aspects of process evaluation which would 

7 

9 



Action Research/Process Evaluation 



be of importance in conducting action research: building an awareness of the value of the 
information derived from process evaluation; examining specific techniques of conducting 
process evaluation; and, perhaps one of the most important, tying process evaluation to 
the results of the research conducted. 

The paper has examined four aspects of conducting process evaluation in examining 
a curricular implementation. It has considered the importance of defining the actual 
curriculum as opposed to the intended curriculum. Next, the level of student involvement 
in the curriculum and tracking it has been deait with. The process of instruction, the use 
of assessment in it, and the relation of both to process evaluation have been examined. 
Finally, tracking time by instructional element through process evaluation has been 
considered. 

Each of these elements forms an important aspect of instruction. In turn, each must 
be examined in a carefully implemented process evaluation. The wealth of information 
resulting from the evaluation will more than repay the practitioner involved in action 
research for the considerable effort necessary to conduct process evaluation. 
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